Interfacing application programs and motion sensors of a device

ABSTRACT

Interfacing application programs and motion sensors of a device. In one aspect, a high-level command is received from an application program running on a motion sensing device, where the application program implements one of multiple different types of applications available for use on the device. The high-level command requests high-level information derived from the output of motion sensors of the device that include rotational motion sensors and linear motion sensors. The command is translated to cause low-level processing of motion sensor data output by the motion sensors, the low-level processing following requirements of the type of application and determining the high-level information in response to the command. The application program is ignorant of the low-level processing, and the high-level information is provided to the application program.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/022,143, filed Jan. 18, 2008, entitled, “Motion Sensing ApplicationInterface”, and is related to co-pending U.S. Utility application Ser.No. 11/774,488 entitled “Integrated Motion Processing Unit (MPU) WithMEMS Inertial Sensing And Embedded Digital Electronics”, filed Jul. 6,2007, all of which is incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to motion sensing devices, andmore specifically to interfacing application programs to motion sensorsof a device.

BACKGROUND OF THE INVENTION

Motion sensors, such as accelerometers or gyroscopes, are often used inelectronic devices. Accelerometers can be used for measuring linearacceleration and gyroscopes can be used for measuring angular velocity.The markets for motion sensors include mobile phones, video gamecontrollers, PDAs, mobile internet devices (MIDs), personal navigationaldevices (PNDs), digital still cameras, digital video cameras, and manymore. For example, cell phones may use accelerometers to detect the tiltof the device in space, which allows a video picture to be displayed inan orientation corresponding to the tilt. Video game console controllersmay use accelerometers and/or gyroscopes to detect motion of the handcontroller that is used to provide input to a game. Picture and videostabilization is an important feature in even low- or mid-end digitalcameras, where lens or image sensors are shifted to compensate for handjittering measured by a gyroscope. Global positioning system (GPS) andlocation base service (LBS) applications rely on determining an accuratelocation of the device, and motion sensors are often needed when a GPSsignal is attenuated or unavailable, or to enhance the accuracy of GPSlocation finding.

Most existing electronic devices tend to use only the very basic ofmotion sensors, such as an accelerometer with “peak detection” or steadystate measurements. For example, current mobile phones use anaccelerometer to determine tilting of the device, which can bedetermined using a steady state gravity measurement. Such simpledetermination cannot be used in more sophisticated applications using,for example, gyroscopes or other applications having precise timingrequirements. Without a gyroscope included in the device, the tiltingand acceleration of the device is not sensed reliably. This is oftenmisleading in non-gyroscope navigational devices; for example, a“non-gyroscope” dead reckoning feature can misjudge automobile locationfor several blocks while the automobile is stopped at a sloped trafficlight. And since motion of the device is not always linear or parallelto the ground, measurement of several different axes of motion using anaccelerometer or gyroscope is required for greater accuracy.

More sophisticated motion sensors typically are not used in electronicdevices. For example, image stabilization and dead reckoning featuresare both important features for a mobile phone with GPS or ahigher-resolution camera, but there is no available solution in thecurrent mobile phone market. Some attempts have been made for moresophisticated motion sensors in particular applications, such asdetecting motion with certain movements. But most of these efforts havefailed or are not robust enough as a product. This is because the use ofmotion sensors to derive motion is complicated. For example, when usinga gyroscope, it is not trivial to identify the tilting or movement of adevice. Using motion sensors for image stabilization, for sensinglocation, or for other sophisticated applications, requires in-depthunderstanding of motion sensors, which makes motion sensing design verydifficult.

The difficulty of motion sensing design also causes difficulty inporting a design from one system to another system. Most raw data frommotion sensors is gathered and processed at the level of the applicationsoftware running on the device, which does not allow for otherapplications to utilize the same sensor output. Furthermore, motionsensing design is typically very system dependent and intended for onespecific application, which prevents it from being ported to othersystems. For example, image stabilization software is typically verydependent on the particular hardware of the digital camera in which itis used, such as the available picture resolution and range of zoom.

Accordingly, a system and method that provides a simple applicationinterface (API) to be available for different applications, allowingmotion sensor data collection to be more easily defined and used by theuser, and allow easier porting and maintenance of a motion sensingdesign for different hardware requirements, would be desirable in manyapplications.

SUMMARY OF THE INVENTION

The invention of the present application relates to interfacingapplication programs to motion sensors of a device. In one aspect of theinvention, providing an interface for one or more applications providedon a motion sensing device includes receiving a high-level command froman application program running on the motion sensing device, where theapplication program implements one of a plurality of different types ofapplications available for use on the motion sensing device. Thehigh-level command requests high-level information derived from theoutput of motion sensors of the motion sensing device that includerotational motion sensors and linear motion sensors. The high-levelcommand is translated to cause low-level processing of motion sensordata output by the motion sensors, the low-level processing followingrequirements of the type of application and intelligently determiningthe high-level information in response to receiving the high-levelcommand. The application program is ignorant of the low-levelprocessing, and the high-level information is provided to theapplication program.

In another aspect of the invention, a method for providing motion sensordata from motion sensors to an application program running on a motionsensing device, includes sampling motion sensor data output by themotion sensors at a first sampling rate, where the motion sensorsinclude rotational motion sensors and linear motion sensors. The motionsensor data is stored in a buffer, and at least a portion of thebuffered motion sensor data is retrieved for the application program ata second sampling rate required by an application implemented by theapplication program, the second sampling rate being lower than the firstsampling rate.

In another aspect of the invention, a method for processing motionsensor data from motion sensors on a motion sensor device includessampling motion sensor data output by the motion sensors at a firstsampling rate, such that the motion sensor data is used to provideinformation to a first application program implementing a firstapplication on the motion sensor device. The motion sensors includerotational motion sensors and linear motion sensors. The motion sensordata is low-pass filtered to a second sampling rate lower than the firstsampling rate, wherein the filtered motion sensor data is used toprovide information to a second application program implementing asecond application on the motion sensor device different from the firstapplication. The first and second application programs are runningsimultaneously on the motion sensor device, and the first applicationrequires motion sensor data to be sampled at sampling rate greater thanthe second sampling rate.

Aspects of the present invention provide an application programminginterface to be available for applications running on a motion sensingdevice. The interface allows easy development of application programsusing complex motion sensor data in devices having motion sensors,allows application programs to be ported to and from different motionsensing devices having different hardware and operating systems, andprovides easier maintenance of systems. Other aspects of the inventionrelax the sampling rate requirements and reduce the processing for anapplication program to sample information related to sensor data, andallow application programs with different sensor data samplingrequirements to run simultaneously on a motion sensing device.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of a motion sensing device suitable for usewith the present invention;

FIG. 2 is a block diagram of one embodiment of a motion processing unitsuitable for use with the present invention;

FIG. 3 is a block diagram illustrating an example of a softwarehierarchy of the present invention;

FIG. 4 is a block diagram illustrating examples of software layers ofFIG. 3 running on an application processor;

FIG. 5 is a flow diagram illustrating a method of the present inventionof an application program accessing an API implemented on the motionsensing device;

FIG. 6 is a flow diagram illustrating a method of the present inventionof an API interfacing with other software layers implemented on themotion sensing device;

FIG. 7 is a flow diagram illustrating a method of the present inventionfor simultaneously processing data for different applicationsconcurrently running on the motion sensor device;

FIGS. 8A-8D are flow diagrams illustrating methods of the presentinvention for providing buffering to reduce processing of theapplication processor in receiving motion sensor output.

DETAILED DESCRIPTION

The present invention relates generally to motion sensing devices, andmore specifically to interfacing application programs to motion sensorsof a device. The following description is presented to enable one ofordinary skill in the art to make and use the invention and is providedin the context of a patent application and its requirements. Variousmodifications to the preferred embodiment and the generic principles andfeatures described herein will be readily apparent to those skilled inthe art. Thus, the present invention is not intended to be limited tothe embodiment shown but is to be accorded the widest scope consistentwith the principles and features described herein.

To more particularly describe the features of the present invention,please refer to FIGS. 1-8D in conjunction with the discussion below.

FIG. 1 is a block diagram of a motion sensing system or device 10suitable for use with the present invention. Device 10 can be providedas a device or apparatus, such as a portable device that can be moved inspace by a user and its motion and/or orientation in space thereforesensed. Such a portable device can be a mobile phone, personal digitalassistant (PDA), video game player, video game controller, navigationdevice, mobile internet device (MID), personal navigation device (PND),digital still camera, digital video camera, binoculars, telephotolenses, or other portable device, or a combination of one or more ofthese devices. In other embodiments, portions of device 10 can beprovided in a portable or movable device, while other portions can beprovided in a non-portable device such as a desktop computer, electronictabletop device, server computer, etc. which can communicate with themoveable or portable portion, e.g., via network connections.

Device 10 includes an application processor 12, memory 14, interfacedevices 16, a motion processing unit 20, analog sensors 22, and digitalsensors 24. Application processor 12 can be one or more microprocessors,central processing units (CPUs), or other processors which run differentsoftware programs for the device 10. For example, different softwareapplication programs such as menu navigation software, games, camerafunction control, navigation software, and phone or other functionalinterfaces can be provided. In some embodiments, multiple differentapplications can be provided on a single device 10, and in some of thoseembodiments, multiple applications can run simultaneously on the device10. Examples of software running on the application processor aredescribed in greater detail below with respect to FIGS. 3 and 4.

Device 10 also includes components for assisting the applicationprocessor 12, such as memory 14 (RAM, ROM, Flash, etc.) and interfacedevices 16. Interface devices 16 can be any of a variety of differentdevices providing input and/or output to a user, such as a displayscreen, audio speakers, buttons, touch screen, joystick, slider,printer, scanner, camera, etc. Memory 14 and interface devices 16 can becoupled to the application processor 12 by a bus 18.

Device 10 also can include a motion processing unit (MPU™) 20. The MPUis a device including motion sensors that can measure motion of thedevice 10 (or portion thereof) in space. For example, the MPU canmeasure one or more axes of rotation and one or more axes ofacceleration of the device. In some embodiments, the components toperform these functions are integrated in a single package. The MPU 20can communicate motion sensor data to an interface bus 21, to which theapplication processor 12 is also connected. In one embodiment, processor12 is a controller or master of the bus 21. For example, in someembodiments the interface bus 21 can be a digital serial interface busimplemented according to such standards as I2C or Serial PeripheralInterface (SPI) bus. Some embodiments can provide bus 18 as the same busas interface bus 21.

MPU 20 includes motion sensors, including one or more rotational motionsensors 26 and one or more linear motion sensors 28. For example, insome embodiments, the rotational motion sensors are gyroscopes and thelinear motion sensors are accelerometers. Gyroscopes 26 can measure theangular velocity of the device 10 (or portion thereof) housing thegyroscopes 26. From one to three gyroscopes can typically be provided,depending on the motion that is desired to be sensed in a particularembodiment. Accelerometers 28 can measure the linear acceleration of thedevice 10 (or portion thereof) housing the accelerometers 28. From oneto three accelerometers can typically be provided, depending on themotion that is desired to be sensed in a particular embodiment. Forexample, if three gyroscopes 26 and three accelerometers 28 are used,then a 6-axis sensing device is provided providing sensing in all sixdegrees of freedom.

In some embodiments the gyroscopes 26 and/or the accelerometers 28 canbe implemented as MicroElectroMechanical Systems (MEMS). Supportinghardware such as storage registers for the data from motion sensors 26and 28 can also be provided.

In some embodiments, the MPU 20 can also include a hardware processingblock 30. Hardware processing block 30 can include logic or controllersto provide processing of motion sensor data in hardware. For example,motion algorithms, or parts of algorithms, may be implemented by block30 in some embodiments. Some embodiments can include a hardware bufferin the block 30 to store sensor data received from the motion sensors 26and 28. For example, in some embodiments described herein, the hardwarebuffer stores sensor data to be sampled by motion algorithms, asdescribed in greater detail below with respect to FIGS. 8A-8D.

One example of an MPU 20 is described below with reference to FIG. 2.Other examples of an MPU suitable for use with the present invention aredescribed in co-pending U.S. patent application Ser. No. 11/774,488,filed Jul. 6, 2007, entitled, “Integrated Motion Processing Unit (MPU)With MEMS Inertial Sensing and Embedded Digital Electronics,” andincorporated herein by reference in its entirety. Suitableimplementations for MPU 20 in device 10 are available from Invensense,Inc. of Santa Clara, Calif.

The device 10 can also include other types of sensors. Analog sensors 22and digital sensors 24 can be used to provide additional sensor dataabout the environment in which the device 10 is situation. For example,sensors such as a barometer, a compass, a temperature sensor, opticalsensor (such as a camera sensor, infrared sensor, etc.), ultrasonicsensor, radio frequency sensor, or other types of sensors can beprovided. In the example implementation shown, digital sensors 24 canprovide sensor data directly to the interface bus 21, while the analogsensors can be provide sensor data to an analog-to-digital converter(ADC) 34 which supplies the sensor data in digital form to the interfacebus 21. In the example of FIG. 1, the ADC 34 is provided in the MPU 20,such that the ADC 34 can provide the converted digital data to hardwareprocessing 30 of the MPU or to the bus 21. In other embodiments, the ADC34 can be implemented elsewhere in device 10.

FIG. 2 shows one example of an embodiment of motion processing unit 20suitable for use with the inventions described herein. The MPU 20 ofFIG. 2 includes an arithmetic logic unit (ALU) 36, which performsprocessing on sensor data. The ALU 36 can be intelligently controlled byone or more programs stored in and retrieved from program RAM (randomaccess memory) 37. The ALU 36 can control a direct memory access (DMA)block 38, which can read sensor data independently of the ALU 36 orother processing unit, from motion sensors such as gyroscopes 26 andaccelerometers 28 as well as other sensors such as temperature sensor39. Some or all sensors can be provided on the MPU 20 or external to theMPU 20; e.g., the accelerometers 28 are shown in FIG. 2 as external tothe MPU 20. The DMA 38 can also provide interrupts to the ALU regardingthe status of read or write operations. The DMA 38 can provide sensordata read from sensors to a data RAM 40 for storage. The data RAM 40provides data to the ALU 36 for processing, and the ALU 36 providesoutput, including processed data, to the data RAM 40 for storage. Bus 21(also shown in FIG. 1) can be coupled to the outputs of data RAM 40and/or FIFO buffer 42 so that application processor 12 can read the dataread and/or processed by the MPU 20.

A FIFO (first in first out) buffer 42 can be used as a hardware bufferfor storing sensor data which can be accessed by the applicationprocessor 12 over the bus 21. The use of a hardware buffer such asbuffer 42 is described in several embodiments below. For example, amultiplexer 44 can be used to select either the DMA 38 writing rawsensor data to the FIFO buffer 42, or the data RAM 40 writing processeddata to the FIFO buffer 42 (e.g., data processed by the ALU 36).

The MPU 20 as shown in FIG. 2 thus can support one or moreimplementations of processing motion sensor data. For example, the MPU20 can process raw sensor data fully, where programs in the program RAM37 can control the ALU 36 to intelligently process sensor data andprovide high-level data to the application processor 12 and applicationprograms running thereon. Or, raw sensor data can be pre-processed orprocessed partially by the MPU 20 using the ALU 36, where the processeddata can then be retrieved by the application processor 12 foradditional low-level processing on the application processor 12 beforeproviding resulting high-level information to the application programs.Or, raw sensor data can be merely buffered by the MPU 20, where the rawsensor data is retrieved by the application processor 12 for therequired low-level processing into high-level data for the applicationprograms. In some embodiments, different applications or applicationprograms running on the same device 10 can use different ones of theseprocessing methods as is most suitable to the application or program.

FIG. 3 is a block diagram illustrating an example of a softwarehierarchy 50 of the present invention that can be implemented/execute onthe application processor 12 of the motion sensing device 10. The layersof software can be provided on a computer readable medium such aselectronic memory or other storage medium such as hard disk, opticaldisk, etc.

An application software layer 52 includes one or more applicationprograms typically provided by one or more application developers andrun on the application processor 12. An application program implements aparticular application (or portion thereof) (which can also be referredto as a “mode”) on the device 10, e.g., each application is implementedby one or more application programs. The application programs canprovide processing and input/output functions as well as the appropriateuser interface functions specific to their application, such ascontrolling a camera lens and displaying images in a camera application,implementing a game in a game application, outputting directions andcurrent location of the device 10 in a navigation application, etc.

The application software layer 52 communicates with system software 54,which manages the resources of the device, including communicationbetween hardware and software components. The software structure for themotion sensing in embedded system applications can be defined inseparate layers as shown in FIG. 3. Thus, system software 54 includes anapplication programming interface (API) layer 56, a real-time operatingsystem (RTOS) layer 58, a motion algorithm layer 60, and a sensor devicedriver layer 62. Additional layers can also be included as appropriate.

API layer 56 provides a set of abstracted and high-level functions foran application program to use, which greatly simplifies the interfaceand communication between an application program and lower-levelcomponents of the device 10 such as the motion sensors. The API layerfacilitates an interface to the motion sensing engine of the device 10by providing a “motion library” of different low-level functions oralgorithms to use which are related to motion sensor data. A particularAPI within the API layer 56 can be defined to correspond to one or moremotion algorithms, where those corresponding algorithm(s) can be used byan application accessing that API.

The API layer 56 can provide predefined, high-level, abstracted commandsas well as associated control parameters or settings to be used by anapplication program to receive information, including high-levelinformation, based on motion sensor data sampled from the motion sensorsof the device 10. The API layer 56 translates a high-level command fromthe application program to the necessary motion algorithms needed toimplement the command, and can thus translate a request for informationrelated to a particular application, to the processing necessary forthat particular application. Each type of application running on thedevice 10 requires a particular set of motion-based high-level databased on the functions of that application type. For example, a usermotion/cursor control type of application may require 2-D or 3-Dcoordinates based on motion of device 10, while an image stabilizationtype of application may require an indication of blur and relatedinformation based on motion of device 10. The motion algorithms the APIaccesses know the precise requirements associated with the low-levelprocessing of motion sensor data to obtain a particular type ofhigh-level information that was commanded by the application program, asrelated to the application associated with the high-level command. Forexample, the motion algorithms know to sample motion sensor data fromthe motion sensors at a higher sampling rate, such as 500 Hz, for animage stabilization application, after having received a high-levelcommand that indicates it will be used for the image stabilizationapplication or function.

Thus the low-level processing can intelligently determine high-levelinformation required by the application program and provide it to theapplication programs, while the application program can be ignorant ofany required low-level processing of motion sensor data, and processingand timing requirements used in that processing, needed to obtain thehigh-level information. For example, an application program developerneed only make a call with an API command to, for example, receiveprocessed motion sensor data appropriate to the developer's application,without having to know how to collect and process raw sensor data. TheAPI layer 56 can be defined as operation system independent as possibleto make application software as portable as possible.

The API layer 56 can provide multiple particular types of APIs that areuseful for one or more types of applications that can be implemented andused on the device 10. For example, a particular motion sensing type ofAPI can be provided, which may be used by certain applications such ascell phone or gaming applications. Multiple different types ofapplications can preferably be implemented on the device 10, andmultiple different APIs for these applications are provided. Some typesof applications may have no need for a particular type of API. Forexample, a navigation application may not need to use an imagestabilization API.

In some embodiments, the API can be broken down into a low-level API(e.g., raw data acquisition, calibrated data acquisition, initializingsensors, setting parameters, performing mathematical functions), amid-level API (e.g., getting processed data such as roll, pitch, andyaw, heading, location, or blur amount), and a high-level API (e.g.,running a DR Kalman filter in one line of code). A mid-level API may beused by application developers that wish to construct their ownapplication from more basic algorithmic building blocks. Otherembodiments can provide only two of the API layer levels. The APIsoftware runs efficiently in the background during system operation, andthe application programs in layer 52 can be provided in any of severaldifferent computer languages or formats. Thus, “bindings” can be createdfor the API layer 56 that allow it to be used with other programminglanguages.

RTOS layer 58 can be provided as an operating system for the device 10to control and manage system resources in real time and enable functionsof the application software 52 as well as other layers such as the APIlayer 56, motion algorithm layer 60, and motion sensor device drivers62. The operating system can interface application programs with othersoftware and functions of the device 10. RTOS layer 58 can be any of avariety of available operating systems suited for embedded systems withreal time operation. The RTOS layer 58 can communicate with any of theother layers, including the application software 52 which uses functionsprovided by the RTOS layer 58. RTOS layer 58 can be implemented as aspecific operating system for the device 10, or a more general operatingsystem adapted for use with the device 10.

Motion algorithm layer 60 provides a “motion library” of motionalgorithms that provide intelligent, lower-level processing for rawsensor data provided from the motion sensors and other sensors. Suchalgorithms in layer 60 can take the raw sensor data and combine and formthe data into high-level information, and/or application-specificinformation.

For example, a motion algorithm can take motion sensor data from severalmotion sensors and process and/or combine the data into high-levelinformation useful for a particular application, such as roll/pitch/yawof the device, or X and Y coordinates for positioning a cursor on atwo-dimensional screen. An algorithm typically has precise timingrequirements for sampling the motion sensor data from the motionsensors. For example, an algorithm may accumulate motion sensor data andintegrate the data points to obtain an angle of motion or other motioncharacteristic of the device 10. In addition, the algorithms can usedata from one or more other sensors, in addition to motion sensor data,to determine the high-level information, such as from a pressure,temperature, and/or direction sensor. The output of the algorithms canbe reported to or retrieved by the API layer 56 which provides thehigh-level information to an application program, and so the processingperformed by the motion algorithms need not be known by any applicationdeveloper. The developer need only use the API functions provided in theAPI layer 56 to request desired high-level information based on motionsensor data.

A motion algorithm can be associated with one or more particularapplications and/or APIs. For example, a “motion sensing” API can use agesture recognition motion algorithm that checks a sequence of sensordata and outputs a determined “feature” such as a gesture when such afeature is detected. This gesture detection is in turn defined in theAPI layer 56 as a simple command, input or event for the applicationsoftware. The motion algorithm can be designed as operation systemindependent as possible, which reduces obstacles in porting thealgorithms between different systems.

The sensor device driver layer 62 provides a software interface to thehardware sensors of the device 10. The driver layer 62 provides aninterface with multiple types of sensors of the device 10, including thegyroscopes 26, accelerometers 28, and other sensors 22 and 24 asdescribed with reference to FIG. 1. For example, the sensed results frommotion sensors 26 and 28 can be written into registers in the MPU by thehardware, and those registers are then accessed by the device driverlayer 62. The driver layer 62 can then provide the accessed sensor datato one or more motion algorithms in the motion algorithm layer 60 forprocessing. In some embodiments, the device driver layer 62 interfaceswith the hardware processing 30 of the MPU, where the hardwareprocessing may have processed some sensor data using motion algorithms.

FIG. 4 is a block diagram illustrating examples of software layers ofFIG. 3 running on the application processor 12. The application softwarelayer 52 can include multiple different application programs toimplement a variety of different types of applications. For example, amobile phone application can be implemented by one or more applicationprograms 60 for implementing phone functions (calling other phones,receiving calls and providing them to the user, etc.), as well as menunavigation. One or more game applications can be implemented byapplication programs 62 that provide different video games, such asdisplaying graphical images on a two-dimensional display screen of thedevice 10 and outputting sound from audio speakers of the device 10, andchanging the output based on user input via buttons, knobs, joystick,stylus, touch pad or touchscreen, microphone, or other input device. Ahealth application can be implemented by health and pedometerapplication programs 64 for controlling and displaying a pedometermeasuring the walking distance of a user, and measuring other bodycharacteristics. A digital camera application can be implemented byapplication programs 66 for controlling a digital still camera ordigital video camera, including image stabilization, zoom and resolutionfunctions, etc. A navigation application can be implemented byapplication programs 68 for personal navigational devices (PNDs),pedestrian dead reckoning (PDR), or other navigational functions, andcan include application programs to control these devices and outputnavigational results to the user, such as current location.

The RTOS layer 58 can oversee the other software layers including theAPI layer 56, the motion algorithms layer 60, and the sensor devicedriver layer 62. The API layer 56 can include different types of APIs,each API characterized for a specific type of high-level informationthat may be useful to one or more application programs that can beexecuted by the device 10. In the example of FIG. 4, the API layer 56includes a motion sensing API 70, an image stabilization API 72, and anavigation API 74.

The motion sensing API 70 can include high-level commands and functionsfor requesting high-level information that describes motion of thedevice 10 (or a portion thereof) or describes features determined by themotion of the device. Such high-level functions can include gesturerecognition, which recognizes particular motions input by a user toinvoke a particular command. For example, a gesture can include movingthe device 10 in a particular direction/orientation or in a sequence ofdirections/orientations, and/or for a specified duration, to initiate afunction associated with that gesture, such as “delete,” “pan,” “zoom,”etc. The API 70 can allow the application program developer to simplyrequest whether any of particular predefined gestures have beendetected, without the developer having to know how to check for thosegestures using raw motion sensor data or how to process that sensor datausing appropriate algorithms. Other motion sensor applications includerequesting motion data relating to user interface functions such asscrolling a view or cursor control, and requesting data indicating inputhas been provided to a game, such as motion/orientation data or agesture to move a graphical object on a screen or activate a functionwithin the game, such as firing a weapon or opening a door.

Another example of a different API in the API layer 56 is an imagestabilization API 72. This API allows an application program to requeststatus of the device 10 relating to high-level image stabilizationfunctions, e.g., as used in a digital still camera or video camera. Forexample, such high-level image stabilization functions can includeestimating the amount or degree of blur within an image that has beencaptured by the camera lens based on the amount of particular motionsdetected during image capture. The API 72 can allow an applicationprogram to simply request the amount of blur in an image, where theapplication program does not have to compute this blur using raw sensordata or process the raw sensor data with algorithms. Other high-levelimage stabilization functions can include image reconstruction, in whichthe application program can request digital processing on a capturedimage to correct or reconstruct the original color or othercharacteristics of a blurred image, without having to perform thatdigital processing itself. Other image stabilization functions caninclude generating a point spread function (PSF), blending images, orevaluating an image for particular characteristics related to motionsensing (such as blur).

Another example of a different API in the API layer 56 is a navigationAPI 74. This API allows an application program to request status of thedevice 10 relating to high-level navigation functions, e.g., as usedwith navigational devices. For example, such high-level navigationfunctions can include providing a dead reckoning location of the device10, and compensating for gravity in such a dead-reckoning estimation.The API 74 can allow an application program to simply request thatgravity be compensated for in the data, rather than having to performthis compensation itself. Other high-level navigation functions caninclude applying a navigation Kalman filter to the sensor data, tocompensate for error in recorded position when providingcontinuously-updated information about the position and/or velocity ofthe device 10, and setting a noise matrix or a process matrix.

The different APIs in the API layer 56 can all be provided on a singledevice 10 that implements the different multiple applications in theapplication layer 52. The ease of using the APIs allows a large andquite different variety of applications to be provided on a singledevice 10 without each application developer being required to havespecialized knowledge in the collection and processing of motion sensordata for a particular application. In some embodiments, the multipleapplications can be executed simultaneously on the device 10. This isdescribed in greater detail below with respect to FIG. 7.

The motion algorithms layer 60 can include algorithms, such as filters,which are called by the APIs in the API layer 56. Some of the motionalgorithms can be basic functional algorithms, such as providing thecurrent orientation of the device 30 in space, or indicating whether thedevice is currently being moved in space or not. Algorithms can combinethe outputs of different sensors and process the results (e.g. usingintegration, derivatives, etc.) to deliver higher accuracy sensingresults, as well as providing higher level information derived from theraw sensor data.

More specific algorithms can also be included for particularapplications. For example, the motion algorithm programs can includegesture recognition algorithms 80 for use with the motion sensing API70, which determine whether particular gestures have been made by a userwith an input device. Such algorithms can determine if the user input a“tapping” gesture or other gesture based on sensor data, etc. or, for amore “mid-level” API, provide roll, pitch and yaw of the device orz-rotation of the device. Similarly, cursor control algorithms 82 can beused with the motion sensing API 70 to determine the X-Y coordinates fora user input device, at which location a cursor is to be displayed on adisplay screen, based on roll, pitch and yaw data as determined by thealgorithm from raw motion sensor data. Other user interface andcontinuous control algorithms can also be used, such as for determiningwhether icons on a display screen have been selected by a user based onsensor input, compensation for hand jitter during cursor control, andmoving one or more displayed images based on motion sensor data. Gamingalgorithms 84 can be used with the motion sensing API 70 to provide datarelated to a gaming application, such as whether a particular gesture ormotion was received which the application uses as input to control agame function.

Image stabilization algorithms can include blur estimation and imagereconstruction algorithms. Blur algorithms 86 can be used to determinewhether a blur has occurred in an image based on motion sensor data atthe time of image capture, such as an image captured by a camera lens.For example, a point spread function (PSF) can be used by the algorithmto determine the blurriness. Image reconstruction algorithms 88 can beused to correct or reconstruct characteristics of an image that may havebeen distorted or removed, e.g., through blur. Other image stabilizationalgorithms are described in copending patent application U.S. patentapplication Ser. No. 11/774,488, and incorporated herein by reference inits entirety. Navigation algorithms can include dead reckoning gravitycompensation algorithms 90 that can be used to compensate for gravitywhen determining a dead reckoning, and a navigation Kalman filter 92that can be used to compensate for errors in sensed positions. Forexample, a dead reckoning location algorithm can use data from one ormore of the accelerometers to determine position of the device 10 anddata from one or more of the gyroscopes 26 to determine the heading ordirection of motion of the device 10.

The device driver layer 62 provides a software interface to the hardwaremotion sensors 26 and 28 of the device 10 and other sensors 22 and 24,as described above with respect to FIG. 3. The layer 62 can includedrivers for gyroscopes 26, accelerometers 28, barometer, compass sensor,and other sensors as described with reference to FIG. 1.

FIG. 5 is a flow diagram illustrating a method 200 of the presentinvention of an application program accessing an API implemented on themotion sensing device 10. The method 200 and the other methods describedherein, if not specifically described, can be implemented in hardware,software, or a combination of both hardware and software. The methodsherein can be implemented using program instructions or program codethat can be executed by a processor or computer and can be provided on acomputer readable medium, such as electronic memory, magnetic tape,magnetic disk, optical disk, etc.

In the method 200, an application program interfaces with a particularAPI of the API layer 56 which provides motion sensing functions usefulfor the application program. In some embodiments, multiple applicationscan interface with a single API, while other application programs mayhave a dedicated API for their exclusive use. An application program canalso access multiple types of APIs, as necessary. The interfacing withthe API is at a high level that allows the application program to ignorethe implementation details of the information it requests. Thus, forexample, a command to the API can be an equivalent to “provide thecurrent location of the device” or “provide the blur kernel for the lastimage” or “provide the type of movement the device is exhibitingcurrently.”

An API of the present invention can also use “event driven programming.”For example, when using a motion sensing API, the application programdoes not typically need to check the input of the motion sensors every10 ms. Instead, if the application program is looking for a gesturecomposed of particular motion sensor data, it requests the API tointerrupt the application program when that gesture is detected (this isan “event”). Examples of the use of an API of the present invention byan application program is shown below in Listings 1 and 2, where “ml”stands for “motion library”).

LISTING 1 mlInitializeSensors( ); mlRegisterGestures (ML_DOUBLE_TAP |ML_ROLL); mlGestureCallBack (got Gesture); ... void gotGesture(gesture){   if (gesture == ML_DOUBLE_TAP) {   [...If the double tap gesture isdetected, perform the functions or operations listed here.]   }   elseif (gesture==ML_ROLL) {   [...If the roll gesture is detected, performthe functions or operations listed here.]   } }

LISTING 2 ml.Bind(ml.EVT_DOUBLE_TAP, OnDoubleTap) // when a double tapgesture is detected, run the function “OnDoubleTap”ml.Bind(ml.EVT_ORIENTATION_CHANGE, OnOrientationChange) // when theorientation is changed, run the function “OnOrientationChange”ml.Bind(ml.EVT_MOTION, OnMotion) // when the portable device moves, runthe function “OnMotion” ml.MainLoop( ) // Begin an infinite loop,running the engine in the background. Don't run any more of the programunless an event requires it. def OnDoubleTap( ):   [...If the double tapgesture is detected, perform the functions or operations listed here.]def OnOrientationChange( ):   [...If an orientation change is detected,perform the functions or operations listed here.] def OnMotion( ):  [...If movement is detected, perform the functions or operations  listed here.]

Method 200 of FIG. 5 starts at 202, and in step 204, the applicationprogram initially sends commands to the API to provide settings that setup or “register” one or more algorithms (such as filters) to run in thebackground as “callback functions” during operation of the device 10.The settings can specify which types of information the applicationprogram wants during system operation. Such settings indicate to the APIwhich algorithms should run and provide information (such as positions,orientations, notifications, etc.) back to the application program. Forexample, the application program may wish to be continually updated withthe position and orientation of the device 10 as it is being movedaround in space by the user. The application program can send settingsto the API that will set up an algorithm to provide the desiredinformation to the application program continually. In some embodimentsthe application program can optionally specify parameters to customizethe high level information it will receive, such as setting the rate atwhich the application program will receive the information from the API,the amount of time during which such information will be sent to theapplication program, the number of times it should be notified, etc. Inaddition, the application program can send settings that indicate one ormore conditions that cause information to be sent to the applicationprogram when those conditions are met as determined based on sensordata. For example, the settings can specify which particular motions orgestures will, when detected, trigger the API to notify the applicationprogram of their occurrence, or the amount of time required between twodetected taps for it to be considered a “double tap” gesture. Theapplication program may also specify the algorithms or filters which thedata is to be processed with, or details such as specifying the integralof continuous motion and/or angle that the device is rotated through tocause a trigger.

In step 206, the application program receives information from the APIbased on the callback function algorithms that were requested to be runin the background, including high-level information. The API providesthe desired information to the application program based on theoperation of appropriate algorithms that were started based on theinformation of step 204 and are running in the background. Step 206 canbe performed continuously, according to any requirements specified bythe application program or inherent to the request as implemented by theAPI and lower layers. For example, in a camera application, theapplication program may need to continuously know if there is excesshand jitter on the camera, where the application program can issuewarning signals to the user if such is the case. Step 206 may also beperformed at irregular intervals, such as when conditions (as previouslyspecified by the application program) are met. For example, theapplication program might need to issue output to the user if the device10 is tilted too far in a certain direction, and would need to benotified of such a tilt. Step 206 may be performed during theperformance of other steps of method 200.

In step 208, the application program checks whether it needs informationto perform its execution, such information being based (at least inpart) on sensor data read by motion sensors of the device 10. Forexample, the application program may need to know the current positionof the device in response to a request for device location from theuser. If no information is required in step 208, then the processreturns to step 206 to receive information (if appropriate). Ifinformation is needed, then in step 210 the application program requeststhe needed high-level information from the API using a high-levelcommand as defined by the API. In step 212, the application programreceives the requested information from the API. The process thenreturns to step 206.

FIG. 6 is a flow diagram illustrating a method 250 of the presentinvention of an API interfacing with other software layers implementedon the motion sensing device 10. The process starts at 252, and in step254, the API receives high-level commands from an application program toset up running algorithms. The high-level commands provide settings thatindicate which high-level information the application program desires toreceive from the running algorithms based (at least in part) on motionsensor data provided by the sensors of the device 10. As explained abovewith reference to FIG. 5, the settings may specify which types ofinformation, the rate at which information is to be sent, conditionsunder which the information is to be sent, etc. The API translates thecommands and settings from the application program into particularmotion algorithms that will provide the requested information followingthe precise requirements of the associated application, e.g. byprocessing motion sensor data sampled from appropriate sensors at therequired sampling rate for that application.

In step 256, the API starts the requested motion algorithms based on thereceived commands and settings, and these algorithms run in thebackground during operation of the device 10. The API can run theparticular algorithms which satisfies the request and settings of theapplication program. In some embodiments, the command from theapplication program is high-level enough that it does not specify theparticular algorithms to run in the background, and the API canassociate the proper algorithms to the received commands. In alternateembodiments, the received commands can specify particular algorithms.The algorithms can sample and process sensor data at the precise timingrequired for the application and report derived information to theapplication program at the required timing of the application program.The algorithms can also check any conditions specified by theapplication program. In some embodiments, a higher software layer, suchas in the API, can check for some or all such specified conditions.

In step 258, the API provides continuous information to the applicationprogram, if any such information was requested, which includeshigh-level information. For example, the API can receive processedinformation from one or more of the algorithms in motion algorithm layer60, and provide this data to the application program. The API oralgorithms preferably provide the information, or allow the applicationprogram to retrieve and sample the information, at a particular rateappropriate to the application implemented by the application program.

In step 260, the API checks to see if it receives a request from theapplication program for particular information. If not, step 264 isinitiated, described below. If so, in step 262, the API provides therequested information to the application program. This information wasobtained by querying the appropriate algorithm and receiving informationfrom the algorithm. The process then returns to step 258. In step 264,the API checks to see if an event has occurred which triggers thesending of information to the application program. For example, theevent can be one or more conditions as specified by the applicationprogram or API being met. If no event has occurred, the process returnsto step 258. If an event has occurred, the API provides the triggeredinformation (typically high-level information) to the applicationprogram in step 266. This information was obtained from the appropriatealgorithm(s). The process then returns to step 258.

In some embodiments, the low-level processing of the motion algorithmsincludes intelligent processing particular to the type of application asindicated by the received high-level command and parameters orinstructions sent by the API to the motion algorithms. A “Content Aware”API can thus be provided, which is intelligent enough to know whatapplication(s) of the device 10 are currently running and enables keyfeatures associated with those applications accordingly. For example, ifthe current running application in use is a gaming application ornavigation application, the API enables appropriate features related togaming or navigation, respectively, including the appropriate sensors toread and timing requirements for reading those sensors, and theappropriate high-level information and functions to provide, includinginterface functions and outputs to the user.

For example, the low-level processing can include examining all of theoutputs of all of the motion sensors of the motion sensing device todetermine the high-level information needed for the type of applicationimplemented by the application program. Depending on the type ofapplication, the low-level processing can selectively use only a subsetof the axes of the motion sensors needed for the type of applicationafter examining all of the motion sensor outputs. For example, in aparticular application running on a 6-axis device 10, an algorithm canexamine all three gyroscopes and all three accelerometers to determinethat the device 10 is either being held vertically in space, in whichcase only sensed data from the accelerometers are to be processed tocreate high level information for that application; or the device isbeing held horizontally in space, in which case only sensed data fromthe gyroscopes are to be processed for that application. In otherembodiments, the high-level command received by the API can determinewhich axes and outputs of the motion sensors to selectively use indetermining the high-level information, based on the type ofapplication. In some embodiments, the high-level command need notspecify the particular axes and outputs to use; rather, the low-levelprocessing can know which sensor outputs to use based on what theparticular high-level command is and the processing the API associateswith that high-level command.

In some embodiments, the selective use of only some of the axes oroutputs of the motion sensors can be further exploited by turning offthe operation or sensing function of the motion sensors (or particularaxes thereof) that are not being used for any currently-runningapplication. This can save power consumption and/or processing bandwidthof the device 10.

Further power management features can also be included in the API layer,including a sleep mode and wake up mode, as well as overall powermanagement for motion sensor resources. For example, the API layer canimplement one or programs running on the application processor 12 or MPU20 that turn off power to gyroscopes 26, which may consume more powerthan other sensors, any time they are not sensing any data, or if theirsensed data is not in use. In one embodiment, the gyroscopes can be“woken up” as soon as some motion of the device 10 is sensed by othermotion sensors.

Examples of API commands which can be used with an API according to thepresent invention are provided below.

mlInitializeSensors( )mlGetRawData( )mlGetCalibratedData( )mlGetAugmentedData( )mlGetHumanMotion( )mlGetRotationMatrix( )mlGetQuaternion( )mIGetRollmlGetPitch( )mIGetYaw( )mlHeading( )mIGetLocation( )mlGetPSF( )mIGetBlurAmount( )mlCalibrate( )mlUpdateState( )mlUpdateRotationMatrix( )mlUpdateHumanMotion( )mlUpdateGesture( )mlRegisterGestures( )mlGestureCallBack( )mlUpdateKalmanFilter( )mlUpdateSensorCalibration( )mlSetNoiseMatrix( )mlSetProcessMatrix( )mlGeneratePSF( )mlBlendImages( )mlEvaluateImage( )

Processing of Simultaneous Multiple Applications

FIG. 7 is a flow diagram illustrating a method 300 of the presentinvention for simultaneously processing data for different applicationssimultaneously running (i.e., being simultaneously implemented) on themotion sensor device 10, by one or more concurrently-running applicationprograms. The method 300 assumes that multiple applications areconcurrently running and require motion sensor data or data derived fromthe motion sensor data. In some embodiments, the method 300 can beimplemented by the motion algorithm layer 60.

At least two of the concurrently-running applications require motionsensor data to be sampled at different minimum sampling rates. Forexample, an image stabilization application requires motion sensor datato be sampled at a high rate, such as 500 Hz, while a user interfaceapplication may require motion sensor data to be sampled at a lowerrate, such as 50 Hz.

The process starts at 302, and in step 304, sensor data is received at afirst sampling rate. The first sampling rate is determined as thehighest minimum sampling rate required by any application currentlyrunning on the device 10 (a sampling rate greater than the minimumrequired can also be used). The sampling rates of other applicationprograms that are available on the system 10, but are not currentlyrunning, are not relevant to the determination of this first samplingrate. In step 306, the sensor data sampled at the first (highest)sampling rate is used for a first application running on the device 10(or used for multiple first applications, if more than one applicationrequires data to be minimally sampled at the first sampling rate). Asdescribed above, motion algorithms in layer 60 can process the sensordata and provide processed high-level information to the API and a firstapplication program implementing the first application.

In step 308, a low-pass filter is used on the sensor data received instep 304 at the first sampling rate to achieve a lower, second samplingrate that is appropriate for a different, second applicationconcurrently running on the device 10, but which is too low of a rate tobe used for the first application(s). For example, the secondapplication may only need sensor data sampled at a lower rate than thefirst application, such that the first sampling rate can be reducedusing the filter. The low pass filter of step 308 can be any of avariety of different types, such as a point running average, an infiniteimpulse response (IIR) filter, a finite impulse response (FIR) filter,etc. In step 310, the sensor data sampled at the second sampling rate isused for the second application (or used for multiple applications, ifmore than one can use sensor data at the second sampling rate). Steps310 and 306 can performed simultaneously to the other steps of themethod 300.

If additional applications are currently running and can use data at adifferent sampling rate, then in step 312, a low-pass filter issimilarly used on the filtered sensor data to achieve even lowersampling rates for additional applications currently running, ifappropriate, and the sampled data is used for those additionalapplications. If such applications are running, the second sampling rateof step 308 is the second-highest sampling rate required by anyapplication program currently running, and the sampling rates of step312 are lower than the second sampling rate of step 308. For example, iftwo additional applications can use different, lower sampling rates,then the data provided at the second sampling rate is filtered toprovide a third sampling rate for the appropriate one of theapplications, and the data at the third sampling rate is filtered toprovide a fourth, lower sampling rate for the other application. Theprocess is then complete at 314.

Method 300 thus allows different applications with different samplingrate requirements to be executed simultaneously on a device 10, which isnormally a difficult task, by low-pass-filtering the data in a treeformat. For example, a blur calculation may require sensor data to besampled at 500 Hz, a user interface application may require a samplingrate of 100 Hz, and a navigation application may require a sampling rateof 20 Hz. At the application program level, the user interface displaymay be updated at 10 Hz, while the navigation data is merged with GPSdata at 1 Hz. A motion sensing engine simultaneously running theseapplications must be able to receive motion sensor data at differentsampling rates simultaneously in order to process sensor data for themultiple applications simultaneously. For example, a user may want totake a picture without blur, while controlling the camera user interfacewith motion gestures. Simultaneously, the device can be dead reckoning,so that the location of the device is not lost during the cameraapplication. For a GPS-enabled camera-phone, a user may not want to losehis or her location each time a picture is taken.

The filtering in the tree format described in method 300 allowsdifferent sampling rates to be extracted from a single highest samplingrate. For example, raw sensor data can be sampled at 500 Hz with a 250Hz anti-aliasing filter for electronic image stabilization. A 10 pointrunning average (e.g., one type of low pass filter that can be used) ofstep 308 provides more anti-aliasing, reducing the bandwidth from 500 Hzto 50 Hz, allowing a 100 Hz sampling rate for a user interfaceapplication (an application can be allowed a sampling rate up to doublethe sensor sampling rate). Another 10 point running average can be usedat 100 Hz, providing a 10 Hz bandwidth suitable for a 20 Hz navigationfilter. The outputs of all these filters can be updated at theirrespective sample rates, without the user's knowledge. This minimizesthe application processor time, since only the image stabilizationfilter needs to be run at the highest sample rates.

In another example, an image stabilization application can be combinedwith a user interface application. Either raw accelerometer data oraugmented data (in this example, gravitational acceleration derived fromsensor data from both accelerometers 28 and gyroscopes 26) can be usedto track the orientation of the camera, used for image rotation(portrait or landscape) and leveling information. Simultaneously,gestures can also be input by the user to control camera modes, such as,for example, switching from still camera mode to video camera mode.Using the present invention, the high-sample-rate gyroscope sensor dataused for blur data is captured at the same time as the lower-sample-rategyroscope sensor data used for gesture recognition data.

In yet another example, navigation can be combined with a user interfaceapplication. In addition to running a dead reckoning algorithm for anavigation application, a handheld device may also be required to run amap rotation algorithm and a user input algorithm for a user interfaceapplication. Using the present invention, these algorithms can berunning simultaneously at different sample rates, since the map rotationrequires low latency in order to be an effective user interface, but thenavigation algorithm may only be required to run at 20 Hz.

The different sampling rates provided by the present invention can alsobe used for power management of the device 10. In some embodiments,power consumption of the device 10 can be reduced, which can beimportant for handheld and portable applications such as pedestriannavigation. For example, in a navigation application, a GPS unit (notshown) can be an additional unit connected to the device 10 to determinethe location of the device. A dead reckoning algorithm in layer 60, usedin combination with a GPS receiver, can more accurately determinecurrent location than if the GPS receiver were solely used to determinelocation. The dead reckoning filter has the additional benefit that itcan reduce the processing required by the GPS unit, thus reducing power.If using dead reckoning with GPS location detection, a 1 Hz GPS samplerate may be reduced to 0.1 Hz, or the satellite tracker may track only 3satellites instead of 12 satellites. Any inaccuracies in the GPSalgorithm can be offset by the additional information achieved by thedead reckoning algorithms.

Hardware and Software Buffering

One reason that hardware, such as the MPU 20, is used in device 10 is toreduce the processing burden on the application processor 12 and/orother processors of the device 10. Using only software for capturingmotion sensor data can be inefficient due to processor/CPU overhead,timing constraints, and difficulty for coding or customization andportability. Ideally a motion sensing driver should use no more than 1%of the CPU time, and guaranteeing correct timing may be difficult withina mobile device operating system. However, hardware is not used toperform all sensor functions for several reasons, including increasedcost, and inaccurate and limited features. For example, adding extrahardware processing to the sensor integrated circuit increases the diearea, which may be unnecessary if that same processing can be performedin software on the application processor 12.

Often the precise timing necessary for motion sensing is required forthe input of data to an algorithm, but not the output to a user, i.e.,the timing requirements are more relaxed for the output of anapplication program to the user. For example, in a user interfaceapplication, the motion sensing is often performed at 100 Hz, withprecise timing to capture all the necessary motion. However, the outputto the user by the application program may only be at 10 Hz, withimprecise timing.

Embodiments of the present invention can use hardware and/or software totake advantage of relaxed timing constraints for the application programoutput and thereby reduce processing time of the application processor.Some embodiments can provide more processing with software for thosesystems having a fast processor, while other embodiments can providemore processing with hardware for systems with a slower processor orother processing burdens. In any embodiment, the same API layer 56 canbe used to provide resulting high-level information to applicationprograms, allowing maximum portability to any system.

FIG. 8A is a flow diagram illustrating a first method 400 of the presentinvention for providing buffering to reduce processing of theapplication processor in receiving motion sensor output. Method 400 useshardware buffering to allow the application processor 12 to relax itstiming constraints in obtaining sensor data.

The method starts at 402, and in step 404 the MPU 20 samples motionsensor data at a higher rate and records the sensor data to a hardwarebuffer. For example, the hardware buffer can be in the hardwareprocessing block 30 of the MPU 20 as shown in FIG. 1, or in differentconnected hardware of the device 10. The MPU samples the sensor data atthe high rate and precise timing normally required to capture motiondata for a particular application.

In step 406, the application processor 12 reads all sensor dataaccumulated in the hardware buffer since the last time it read thebuffer, where the processor 12 samples the hardware buffer (and sensordata) at a lower rate than the MPU used to record the sensor data. Theprocessor reads all these sensor data points in the hardware buffer atonce.

In step 408, the motion algorithms are updated in quick succession basedon the retrieved multiple sensor data points, and the final output ofthe algorithms is provided to the application program at the lowersampling rate. This provides information to the application program atthe lower required sampling rate. The process then returns to step 404.

The accumulation of sensor data in the buffer allows applicationprograms of application processor 12 to relax timing constraints forsampling data from the MPU without missing any motion information. Forexample, the MPU hardware may record sensor data in the buffer at thehigher rate of 100 Hz. An application program on application processor12 may need to update a display screen based on motion at only a 10 Hzsample rate. Using the method 400, the motion algorithms can read allthe data stored in the MPU buffer at once at a rate of 10 Hz. Thus, if10 data points are in the buffer, the motion algorithms are updated 10times in quick succession, and the display information output by themotion algorithms is provided to the application program, effectively at10 Hz. This allows the application processor 12 to read the buffer onlywhen it needs to, e.g., at a rate of 10 Hz (the display refresh rate),which reduces the total processing time of the processor 12. Thereduction may not be by a large amount, since the motion algorithms arestill being run at the higher rate on average using all data points ofthe received motion data.

FIG. 8B is a flow diagram illustrating a second method 420 of thepresent invention for providing buffering to reduce processing of theapplication processor in receiving motion sensor output. Method 420 usessoftware buffering to allow the application processor 12 to relax itstiming constraints in obtaining sensor data.

The method starts at 422, and in step 424 an interrupt routine samplesmotion sensor data at a higher rate. The interrupt routine is a softwareroutine running on the application processor 12 and can be provided inthe algorithm layer 60, for example. The interrupt routine samples thesensor data from the MPU 20 at the high rate and with the accurate andprecise timing normally required to capture motion data appropriate foran application. In step 426, the interrupt routine stores the data readin step 424 in a software buffer, without processing the data. Forexample, the software buffer is memory accessible to the applicationprocessor and can be provided in the application processor 12(registers, RAM, etc.), in the memory 14 of the device 10, or in otheravailable memory.

In step 428, the interrupt routine interrupts the processor 12 toindicate that enough sensor data is accumulated to be read. In step 430,the application processor 12 (e.g., a motion algorithm) reads all thesensor data accumulated in the software buffer since the last time itread the buffer. Thus, the processor 12 reads the sensor data at a lowerrate than the interrupt routine used to sample and store the sensordata. In step 432, the motion algorithms are updated in quick successionbased on the retrieved multiple sensor data points, and the final outputof the algorithms is provided to the application program at the lowersampling rate. This provides information to the application program atthe lower required sampling rate. The process then returns to step 424.

As with the hardware buffer embodiment of FIG. 8A, the accumulation ofsensor data in the buffer allows the application processor 12 to relaxits timing constraints with regard to the application program outputwithout missing any motion information. For example, the interruptroutine may sample and store sensor data in the buffer at the higherrate of 100 Hz, but the application processor 12 can read all the datastored in the software buffer at once at a relaxed rate. Thus, at asample rate of 10 Hz, the application processor can retrieve the latest10 data points in the buffer and process the motion algorithm 10 timesin quick succession, allowing a display application program to receiveappropriate information from the algorithms at 10 Hz and provide itsoutput at 10 Hz.

FIG. 8C is a flow diagram illustrating a third method 450 of the presentinvention for providing buffering to reduce processing of theapplication processor in receiving motion sensor output. Method 450 useshardware interrupts to allow the application processor 12 to reduce itstiming constraints in obtaining sensor data.

The method starts at 452, and in step 454 the MPU 20 receives the sensordata, and the sensor data is stored in a buffer. This sampling isperformed at a high rate and with the accurate and precise timingnormally required to capture motion sensor data. The sensor data can bestored in a hardware buffer or in a software buffer.

In step 456, the MPU scans the buffered sensor data. In step 458, theMPU checks whether any “interesting” properties have been detected inthe scanned sensor data. Such properties are patterns or features in thedata that may be relevant to a particular algorithm or application, suchthat they should be reported to the application processor 12. Forexample, the properties can include a predetermined threshold that hasbeen exceeded, or a pattern in the data indicating a particular type ofmotion. For example, when a gesture is made by the user, thresholds maybe exceeded on at least one motion sensor. Or a particular thresholdvalue may have been exceeded indicating a level of jitter on the cameralens portion of device 10. If no interesting properties are detected bythe MPU in step 458, then the process returns to step 454.

If one or more interesting properties is detected, then in step 460 theMPU sends an interrupt to the application processor. This interruptindicates to the application processor that an interesting property hasbeen detected and should be examined by the application processor. Thus,in step 462, the application processor 12 retrieves data stored in abuffer that includes the detected interesting properties, e.g., the MPUcan buffer the interesting property data in a different buffer or otherarea of the buffer, which the processor 12 can read from, or the MPU candesignate the location to read from in the main buffer. The applicationprocessor 12 then analyzes the retrieved data as appropriate to theapplication, e.g., process the data with an algorithm in the algorithmlayer 60 and provide the data via the API 56 to the application programat a reduced sampling rate. The process then returns to step 454.

The use of the MPU hardware to check for particular properties in thesensed data can reduce the processing overhead of the applicationprocessor. Even if the MPU sends interrupts to the application processorat a high rate, such as detecting interesting properties five times persecond, it still reduces the number of times the algorithm processingand application program processing must be run from 100 Hz to 5 Hz. Withthis embodiment, the timing constraints have been reduced for allsoftware on the application processor 12, and the processing time ofprocessor 12 has thus also been reduced.

FIG. 8D is a flow diagram illustrating a fourth method 470 of thepresent invention for providing buffering to reduce processing of theapplication processor in receiving motion sensor output. Method 470 usesa more advanced method of hardware pre-processing to allow theapplication processor 12 to reduce its timing constraints in obtainingsensor data.

The method starts at 472, and in step 474 the MPU 20 receives the sensordata, and the sensor data is stored in a buffer, such as the hardwarebuffer of the MPU. The data is sampled and stored at a high rate andwith the precise timing normally required to capture motion sensor data.In step 476, the MPU pre-processes the sensor data to reduce the data toa set of relevant features for one or more applications. The MPU caninclude motion algorithms similar to, or a subset of, the motionalgorithms present in the motion algorithm layer 60 of the applicationprocessor software. For example, in some embodiments the MPU can detectparticular gestures by processing the sensor data. This processingability can determine which features are relevant to the applicationsrunning on the application processor 12, and/or provide high-levelinformation for an application program. For example, relevant featurescan be parameters calculated from additional derivatives and integralsof the inertial data.

In step 478, the set of relevant features is provided to the applicationprogram on the application processor 12, at a lower sampling rate thanthe sampling rate used to capture the motion sensor data. Thus, forexample, a 100 Hz sample rate for capturing sensor data can be reducedto a 5 Hz sample rate providing only the relevant information to theapplication processor. The relevant features can then be provided to theapplication program at the reduced sampling rate (or to an appropriatemotion algorithm for further processing, which provides information tothe application program at the reduced sampling rate). Similar to theembodiment of FIG. 8C, with this embodiment, the timing constraints havebeen reduced for all software on the application processor 12, and theprocessing time of processor 12 has thus also been reduced.

Although the present invention has been described in accordance with theembodiments shown, one of ordinary skill in the art will readilyrecognize that there could be variations to the embodiments and thosevariations would be within the spirit and scope of the presentinvention. Accordingly, many modifications may be made by one ofordinary skill in the art without departing from the spirit and scope ofthe appended claims.

1. A method for providing an interface for one or more applicationprograms provided on a motion sensing device, the method comprising:receiving a high-level command from an application program running onthe motion sensing device, wherein the application program implementsone of a plurality of different types of applications available for useon the motion sensing device, the high-level command requestinghigh-level information derived from the output of at least one motionsensor of the motion sensing device; translating the high-level commandto cause low-level processing of multiple axes of motion sensor dataoutput by the motion sensors, the low-level processing followingrequirements of the type of application and intelligently determiningthe high-level information from the motion sensor data in response toreceiving the high-level command, wherein the at least one motion sensorincludes at least one rotational motion sensor or at least one linearmotion sensor sensors; and providing the high-level information to theapplication program.
 2. The method of claim 1 wherein the requirementsof the type of application include a precise sampling rate for themotion sensor data obtained from the motion sensors.
 3. The method ofclaim 1 wherein the motion sensor data is obtained from the motionsensors at a precise sampling rate following a requirement of theapplication, and wherein at least two of the plurality of differenttypes of applications available for use on the motion sensing devicehave different sampling rate requirements for the motion sensor data. 4.The method of claim 1 wherein providing the high-level information tothe application program includes providing the high-level informationsuch that it is received by the application program at a sampling rateappropriate to the type of the application implemented by theapplication program.
 5. The method of claim 1 wherein the high-levelinformation is received by the application program at a sampling ratelower than the sampling rate used to sample the motion sensor data. 6.The method of claim 1 wherein a plurality of different applicationprograms that implement the different types of applications availablefor use on the motion sensing device run simultaneously on the motionsensing device.
 7. The method of claim 6 wherein the plurality ofdifferent application programs require the high-level information to beprovided at different sampling rates from each other, and wherein thesampling rate of the motion sensor data for one of the applications isobtained by filtering the sampling rate of the motion sensor data for adifferent one of the applications.
 8. The method of claim 1 wherein theplurality of different types of applications available for use on themotion sensing device include at least one of a motion sensingapplication, an image stabilization application, and a navigationapplication.
 9. The method of claim 1 wherein the processing of themotion sensor data includes running one or more motion algorithms in thebackground during the operation of the motion sensing device, the motionalgorithms checking the motion sensor data for one or more conditions tobe met, which causes the application program to be notified.
 10. Themethod of claim 1 further comprising buffering the motion sensor dataobtained at a higher sampling rate and with precise timing requirements,and processing the buffered motion sensor data and providing thebuffered motion sensor data to the application program at a lowersampling rate and relaxed timing requirements.
 11. The method of claim 1wherein the plurality of rotational motion sensors include threegyroscopes, and wherein the plurality of linear motion sensors includethree accelerometers, the gyroscopes and accelerometers providing 6-axissensing.
 12. The method of claim 1 wherein the high level information isfurther determined from additional sensors of the motion sensing device,the additional sensors including at least one of a temperature sensor, apressure sensor, and a compass.
 13. The method of claim 1 wherein thelow-level processing includes examining all the outputs of all of themotion sensors of the motion sensing device to determine the high-levelinformation.
 14. The method of claim 13 wherein the low-level processingdetermines to selectively use only a subset of the axes of the motionsensors for the type of application after examining all of the outputsof the motion sensors.
 15. The method of claim 14 wherein the low-levelprocessing turns off the operation of motion sensors not used for thetype of application to save power consumption or processing bandwidth.16. The method of claim 1 wherein the low-level processing determineswhich outputs of the motion sensors to selectively use in determiningthe high-level information, based on the particular high-level command.17. The method of claim 1 wherein the low-level processing is performedat least in part by a motion processing unit distinct from and incommunication with an application processor of the motion sensingdevice.
 18. A method for providing motion sensor data from motionsensors on a motion sensing device to an application program running onthe motion sensing device, the method comprising: sampling motion sensordata output by the motion sensors at a first sampling rate, wherein themotion sensors include a plurality of rotational motion sensors and aplurality of linear motion sensors; storing the motion sensor data in abuffer; and retrieving at least a portion of the buffered motion sensordata for the application program at a second sampling rate required byan application implemented by the application program, the secondsampling rate being lower than the first sampling rate.
 19. The methodof claim 18 wherein the buffer is a hardware buffer implemented in amotion processing unit separate from an application processor.
 20. Themethod of claim 18 wherein the buffer is a software buffer implementedin memory, and wherein an interrupt routine samples the motion sensordata and interrupts a application processor running the applicationprogram to retrieve at least a portion of the buffered motion sensordata.
 21. The method of claim 18 wherein a motion processing unitseparate from an application processor scans the stored motion sensordata to detect properties relevant to the application program.
 22. Themethod of claim 18 wherein a motion processing unit separate from anapplication processor reduces the motion sensor data in the buffer to aset of features relevant to the application program, and wherein theretrieved portion of the buffered motion sensor data is the set offeatures.
 23. A method for processing motion sensor data from motionsensors on a motion sensor device, the method comprising: samplingmotion sensor data output by the motion sensors at a first samplingrate, such that the motion sensor data is used to provide information toa first application program implementing a first application on themotion sensor device, wherein the motion sensors include a plurality ofrotational motion sensors and a plurality of linear motion sensors; andlow-pass filtering the motion sensor data to a second sampling ratelower than the first sampling rate, wherein the filtered motion sensordata is used to provide information to a second application programimplementing a second application on the motion sensor device differentfrom the first application, wherein the first and second applicationprograms are running simultaneously on the motion sensor device, andwherein the first application requires motion sensor data to be sampledat sampling rate greater than the second sampling rate.
 24. A computerreadable medium storing program instructions to be executed by acomputer and for providing an interface for one or more applicationprograms provided on a motion sensing device, the program instructionsperforming steps comprising: receiving a high-level command from anapplication program running on the motion sensing device, wherein theapplication program implements one of a plurality of different types ofapplications available for use on the motion sensing device, thehigh-level command requesting high-level information derived from theoutput of a plurality of motion sensors of the motion sensing device;translating the high-level command to cause low-level processing ofmultiple axes of motion sensor data output by the motion sensors, thelow-level processing following requirements of the type of applicationand intelligently determining the high-level information from the motionsensor data in response to receiving the high-level command, wherein themotion sensors include at least one rotational motion sensor and atleast one linear motion sensor; and providing the high-level informationto the application program.
 25. The computer readable medium of claim 24wherein the requirements of the type of application include a precisesampling rate for the motion sensor data obtained from the motionsensors.
 26. The computer readable medium of claim 24 wherein the motionsensor data is obtained from the motion sensors at a precise samplingrate following a requirement of the application, and wherein at leasttwo of the plurality of different types of applications available foruse on the motion sensing device have different sampling raterequirements for the motion sensor data.
 27. The computer readablemedium of claim 24 wherein providing the high-level information to theapplication program includes providing the high-level information suchthat it is received by the application program at a sampling rateappropriate to the type of the application implemented by theapplication program.
 28. The computer readable medium of claim 24wherein a plurality of different application programs that implement thedifferent types of applications available for use on the motion sensingdevice run simultaneously on the motion sensing device.
 29. The computerreadable medium of claim 28 wherein the plurality of differentapplication programs require the high-level information to be providedat different sampling rates from each other, and wherein the samplingrate of the motion sensor data for one of the applications is obtainedby filtering the sampling rate of the motion sensor data for a differentone of the applications.
 30. The computer readable medium of claim 24further comprising buffering the motion sensor data obtained at a highersampling rate and with precise timing requirements, and processing thebuffered motion sensor data and providing the buffered motion sensordata to the application program at a lower sampling rate and relaxedtiming requirements.
 31. A motion sensing device including an interfacefor one or more application programs provided on the motion sensingdevice, the motion sensing device comprising: a motion processing unitincluding a plurality of motion sensors for sensing the motion of themotion sensing device, the motion sensors including at least onerotational motion sensor and at least one linear motion sensor; and anapplication processor coupled to the motion processing unit, theapplication processor providing an application program that implementsone of a plurality of different types of applications available for useon the motion sensing device, and an application interface that receivesa high-level command from the application program, the high-levelcommand requesting high-level information derived from the output of themotion sensors of the motion sensing device, wherein the applicationinterface translates the high-level command to cause low-levelprocessing of motion sensor data output by the motion sensors, thelow-level processing following requirements of the type of applicationand determining the high-level information in response to receiving thehigh-level command, and wherein the application interface provides thehigh-level information to the application program.
 32. A method forproviding an interface for an application program running on a handheldelectronic device, the electronic device comprising a display and a setof motion sensors sensing rotational rate around at least three axes andlinear acceleration along at least three axes, the method comprising:receiving a high-level command from the application program, thehigh-level command requesting high-level motion information; andproviding high-level motion information to the application program,where the high-level motion information is based on motion sensor dataproduced by at least one motion sensor and on the high-level command.33. A method for providing an interface for a plurality ofmotion-responsive application programs running on a handheld electronicdevice, the electronic device comprising a display and a set of motionsensors sensing rotational rate around at least one axis and linearacceleration along at least one axis, each of the motion-responsiveapplication programs capable of utilizing high-level motion information,the method comprising: receiving a high-level command from at least onemotion-responsive application program, the high-level command requestingspecific high-level motion information; and providing high-level motioninformation to the application program, where the high-level motioninformation is based on motion sensor data produced by at least onemotion sensor and on the high-level command.